Assessing the Frequency of Empirical Evaluation in Software Modeling Research

Researchers in software modeling often publish new tools or methodologies that claim to offer some advantage to the modeling community. There are different methods by which those claims can be evaluated. In this paper, we examine the degree to which such claims are supported by various types of empirical evaluation. We surveyed five editions of the MoDELS conference from 2006-2010, as well as the primary conference that focuses on empirical software engineering (the International Symposium on Empirical Software Engineering and Metrics), to understand the frequency with which empirical evaluation has been reported in the software modeling community. Our summary of 266 MoDELS papers found that 195 (73%) of the publications performed no empirical evaluation. This paper summarizes our findings from that survey and offers recommendations for improving the awareness and need for empirical evaluation in software modeling research.